earthly

2022-04-05 · 7 min read

Earthly replaces Makefile+Dockerfile with a Docker-like DSL. You write Make-like build targets, which can have dependencies on other targets. The contents of each build target is (effectively) a Dockerfile.

The ultimate goal is repeatable, reproducible builds. Repeat a failed build from CI. Seamlessly reproduce your developer environment on a colleague's machine. Etc etc...

Earthly vs Nix #

Both use linux namespaces for isolation. Nix has an absolutely awful DSL language that I have to relearn every time I touch it. Earthfiles are significantly more imperative; they look like a list of Makefile targets with Dockerfile recipes.

Earthly vs Buck / Bazel #

Bazel and Buck provide truely hermetic builds, but they each require complete control over the entire toolchain, so no using npm, cargo, etc... This is fine for Google/Facebook, but not for smaller teams (IMO).

Earthly claims to strike a more pragmatic balance between truely repeatable and deterministic builds and productive development for a smaller team.

Docs #

It appears build steps must explicitly mark files as "artifacts". You must then explicitly copy these artifacts over in dependent build steps.
Likewise, output artifacts must be explicitly exported to the user's FS from the container FS.
Exported artifacts are not transitive, so if your dependency explicitly exports to the user's FS, your build step won't unless you also explicitly export.
Can easily push artifacts to remote destinations, run db migrations, cut releases, etc... as a build step.
Like normal Dockerfiles, an intermediate build step is cached as a new layer.

Thoughts #

Pros #

Seems like a solid choice for a CI pipeline and devs occasionally reproducing CI runs locally.
Builds are nicely isolated from the dev machine, reducing implicit state that's actually necessary for builds to succeed.
Builds are (mostly) reproducible across environments. Though, my experience with a big Rust project was that this was not a problem.
I love that build step inputs and outputs are marked explicitly. Following an unfamiliar Earthfile doesn't feel too bad.
Once you've set up a common development base image, all developers can share good quality tools and configuration without have to independently set up debuggers, profilers, etc...
Can use a shared build cache like Bazel, Buck, or scc so beefy builds don't have to take forever. CI can also shared this build cache!

Cons #

Like Docker, toolchains are stuck inside containers, usually in a way that is inaccessible to other dev tools, like IDEs, debuggers, profilers, etc...
I worry that basic dev tools like running an LSP might be challenging to set up? Would this be like mounting the dev directory as a volume for the LSP service? Or is this not even a problem at all... idk. I don't use Docker frequently enough to know.
Some weird idiosyncrasies, presumably due to the Docker layering model. Developers need a solid mental model to avoid committing egregiously large, slow, or uncacheable intermediate layers.
1. Example: explicit caching of derived Cargo dependency state in first three build steps: https://github.com/earthly/earthly/blob/main/examples/rust/Earthfile
Rust has a nice enough runner (cargo) that you don't hit a lot of the reproducibility problems experienced in other languages *cough* C/C++.
Caching is still too coarse-grained compared to running stateful native tools like cargo locally.
Unlike Bazel or Buck, which take full control of the toolchain, it seems more likely to hit non-determinism with Earthly, since we're using language tools which typically don't know or care about reproducibility.
Get to learn yet another fun and exciting DSL, with its own special and unique patterns and other miscellaneous weirdness.
Too slow in tight development loop. On my beefy desktop, a no-op build takes at least 5s. On my M1 MBP, a no-op build takes at least 8-10s. This means any non-trivial build step has a minimum 8-10s duration, which is frankly unacceptable. Non-trivial file copying also adds significant overhead.
Multi-platform builds (targetting amd64/linux) failed on my M1 Mac for inscrutable reasons. Most likely not earthly's fault, but an issue nonetheless.
Feels unwieldy running integration tests that need access to low-level hardware devices. Our requirement is probably not typical though.
Remote builder authentication uses mTLS certs. Provisioning these seems like a pain, esp. when I already have ssh secrets provisioned : (
Overall, Earthly still feels a bit early. I'd check back after another 6mo (that would be Q1 2023 as of writing).

Installation #

# Ubuntu/Debian/PopOS!
$ curl --proto '=https' --tlsv1.3 -sSfL https://pkg.earthly.dev/earthly.pgp \
	| gpg --dearmor \
	| sudo tee /usr/share/keyrings/earthly.gpg
$ echo "deb [arch=amd64 signed-by=/usr/share/keyrings/earthly.gpg] https://pkg.earthly.dev/deb stable main" \
  | sudo tee /etc/apt/sources.list.d/earthly.list > /dev/null
$ sudo apt update
$ sudo apt install earthly

# macOS
$ brew install earthly

Earthly Remote Build #

Install docker on remote (Ubuntu) #

Install docker if you haven't already

$ sudo mkdir -p /etc/apt/keyrings
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
$ echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" \
	| sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

$ sudo apt update
$ sudo apt install docker-ce docker-ce-cli containerd.io \
	docker-compose-plugin

$ sudo groupadd docker
$ sudo usermod -aG docker $USER
$ newgrp docker # activate changes? else log out and back in

# sanity check docker installation
$ docker run --rm hello-world

# start docker on restart
$ sudo systemctl enable docker.service
$ sudo systemctl enable containerd.service

Run earthly/buildkit (remote) #

This runs a modified buildkit listening on TCP port 8372, without authentication.

$ docker run \
	--privileged -t -v earthly-tmp:/tmp/earthly:rw \
	-e BUILDKIT_TCP_TRANSPORT_ENABLED=true -p 8372:8372 \
	earthly/buildkitd:v0.6.15

(FAIL) Run ssh forward (host) #

$ ssh -NL 8372:localhost:8372 phlip9@sgxdev.phlip9.com

FAIL: It appears Earthly tries to do a local build if the buildkit hostname is localhost : /

Open VM port #

Very unsafe. Do this only for brief sanity testing.

$ az vm open-port --name sgxdev2 --port 8372

TODO: provision TLS certs.

Test remote build #

$ EARTHLY_BUILDKIT_HOST=tcp://my-remote-buildkit:8372 earthly +my-cool-target

It works!

Misc. Notes #

Earthfile Syntax #

SAVE ARTIFACT .. AS LOCAL .. + Only save the output locally if we ask for the specific target OR we use the BUILD command
SAVE IMAGE foo:latest + Saves current target layer as a docker image named foo with tag latest
SAVE IMAGE --push .. + Push image to remote repo
Adding --push to a RUN command defines an "external" command. These will only run if the entire build succeeds. You also need to run the earthly build with --push to enable these commands.

release:
    RUN --push --secret GITHUB_TOKEN=+secrets/GH_TOKEN github-release upload

$ earthly --push +release

Also useful for running things like DB migrations

migrate:
    FROM +build
    RUN --push bundle exec rails db:migrate

or terraform apply

apply:
    RUN --push terraform apply -auto-approve

Targets in other Earthfile (relative path, in same repo)

build:
	# ./services/foobar/Earthfile
	#   -> contains `deps` target
	FROM ./services/foobar+deps
	# ..

Import targets from other Earthfile

VERSION 0.6
IMPORT ./services/foobar
# ..

build:
	FROM foobar+deps
	# ..

Run docker commands inside a target using WITH DOCKER .. END + Will init a docker daemon that can be used in a RUN command + Recommend using earthly's "docker-in-docker" (dind) container earthly/dind:alpine
Pulling a docker image from docker hub

hello:
	FROM earthly/dind:alpine
	WITH DOCKER --pull hello-world
		RUN docker run hello-world
	END

Loading an image created by another target

my-hello-world:
	FROM ubuntu
	CMD echo "hello world"
	SAVE IMAGE my-hello:latest

hello:
	FROM earthly/dind:alpine
	WITH DOCKER --load hello:latest=+my-hello-world
		RUN docker run hello:latest
	END